This invention generally relates to autostereoscopic display systems for viewing images and more particularly relates to an apparatus and method for providing autostereoscopic viewing that is capable of adapting to viewer characteristics and response.
The potential value of autostereoscopic display systems is widely appreciated particularly in entertainment and simulation fields. Autostereoscopic display systems include xe2x80x9cimmersionxe2x80x9d systems, intended to provide a realistic viewing experience for an observer by visually surrounding the observer with a 3-D image having a very wide field of view. As differentiated from the larger group of stereoscopic displays that include it, the autostereoscopic display is characterized by the absence of any requirement for a wearable item of any type, such as goggles, headgear, or special glasses, for example. That is, an autostereoscopic display attempts to provide xe2x80x9cnaturalxe2x80x9d viewing conditions for an observer. An acknowledged design goal for immersion systems is to provide the most realistic viewing environment possible. While this relates most pronouncedly to visual perception, it can also encompass auditory, tactile, and other sensory perception as well.
In an article in SID 99 Digest, xe2x80x9cAutostereoscopic Properties of Spherical Panoramic Virtual Displaysxe2x80x9d , G. J. Kintz discloses one approach to providing autostereoscopic display with a wide field of view. Using the Kintz approach, no glasses or headgear are required. However, the observer""s head must be positioned within a rapidly rotating spherical shell having arrays of LED emitters, imaged by a monocentric mirror, to form a collimated virtual image. While the Kintz design provides one solution for a truly autostereoscopic system having a wide field of view, this design has considerable drawbacks. Among the disadvantages of the Kintz design is the requirement that the observer""s head be in close proximity to a rapidly spinning surface. Such an approach requires measures to minimize the likelihood of accident and injury from contact with components on the spinning surface. Even with protective shielding, proximity to a rapidly moving surface could, at the least, cause the observer some apprehension. In addition, use of such a system imposes considerable constraints on head movement, which compromises the illusion of natural reality that is a goal.
One class of autostereoscopic systems operates by imaging the exit pupils of a pair of projectors onto the eyes of an observer, is as outlined in an article by S. A. Bentone, T. E. Slowe, A. B. Kropp, and S. L. Smith (xe2x80x9cMicropolarizer-based multiple-viewer autostereoscopic display,xe2x80x9d in Stereoscopic Displays and Virtual Reality Systems VI, SPIE, January, 1999). Pupil imaging, as outlined by Benton, can be implemented using large lenses or mirrors. An observer whose eyes are coincident with the imaged pupils can view a stereoscopic scene without crosstalk, without wearing eyewear of any kind.
It can be readily appreciated that the value and realistic quality of the viewing experience provided by an autostereoscopic display system using pupil imaging is enhanced by presenting the 3-D image with a wide field of view and large exit pupil. Such a system is most effective for immersive viewing functions if it allows an observer to be comfortably seated, without constraining head movement to within a tight tolerance and without requiring the observer to wear goggles or other device. For fully satisfactory 3-D viewing, such a system should provide separate, high-resolution images to right and left eyes. It can also be readily appreciated that such a system is most favorably designed for compactness, to create an illusion of depth and width of field, while occupying as little actual floor space and volume as is possible.
It is also known that conflict between depth cues associated with vergence and accommodation can adversely impact the viewing experience. Vergence refers to the degree at which the observer""s eyes must be crossed in order to fuse the separate images of an object within the field of view. Vergence decreases, then vanishes as viewed objects become more distant. Accommodation refers to the requirement that the eye lens of the observer change shape to maintain retinal focus for the object of interest. It is known that there can be a temporary degradation of the observer""s depth perception when the observer is exposed for a period of time to mismatched depth cues for vergence and accommodation. It is also known that this negative effect on depth perception can be mitigated when the accommodation cues correspond to distant image position.
An example of a conventional autostereoscopic display unit is disclosed in U.S. Pat. No. 5,671,992 (Richards), at which a seated observer experiences apparent 3-D visual effects created using images generated from separate projectors, one for each eye, and directed to the observer using an imaging system comprising a number of flat mirrors.
Conventional solutions for stereoscopic imaging have addressed some of the challenges noted above, but there is room for improvement. For example, some early stereoscopic systems employed special headwear, goggles, or eyeglasses to provide the 3-D viewing experience. As just one example of such a system, U.S. Pat. No. 6,034,717 (Denting et al.) discloses a projection display system requiring an observer to wear a set of passive polarizing glasses in order to selectively direct the appropriate image to each eye for creating a 3-D effect.
Certainly, there are some situations for which headgear of some kind can be considered appropriate for stereoscopic viewing, such as with simulation applications. For such an application, U.S. Pat. No. 5,572,229 (Fisher) discloses a projection display headgear that provides stereoscopic viewing with a wide field of view. However, where possible, there are advantages to providing autostereoscopic viewing, in which an observer is not required to wear any type of device, as was disclosed in the device of U.S. Pat. No. 5,671,992. It would also be advantageous to allow some degree of freedom for head movement. In contrast, U.S. Pat. No. 5,908,300 (Walker et al.) discloses a hang-gliding simulation system in which an observer""s head is maintained in a fixed position. While such a solution may be tolerable in the limited simulation environment disclosed in the Walker et al. patent, and may simplify the overall optical design of an apparatus, constraint of head movement would be a disadvantage in an immersion system. Notably, the system disclosed in the Walker et al. patent employs a narrow viewing aperture, effectively limiting the field of view. Complex, conventional projection lenses, disposed in an off-axis orientation, are employed in the device disclosed in U.S. Pat. No. 5,908,300, with scaling used to obtain the desired output pupil size.
A number of systems have been developed to provide visual depth effects by presenting to the observer the combined image, through a beamsplitter, of two screens at two different distances from the observer, thereby creating the illusion of stereoscopic imaging, as is disclosed in U.S. Pat. No. 5,255,028 (Biles). However, this type of system is limited to small viewing angles and is, therefore, not suitable for providing an immersive viewing experience. In addition, images displayed using such a system are real images, presented at close proximity to the observer, and thus likely to introduce the vergence/accommodation problems noted above.
It is generally recognized that, in order to minimize vergence/accommodation effects, a 3-D viewing system should display its pair of stereoscopic images, whether real or virtual, at a relatively large distance from the observer. For real images, this means that a large display screen must be employed, preferably placed a good distance from the observer. For virtual images, however, a relatively small curved mirror can be used as is disclosed in U.S. Pat. No. 5,908,300 (Walker). The curved mirror acts as a collimator, providing a virtual image at a large distance from the observer. Another system for stereoscopic imaging is disclosed in xe2x80x9cMembrane Mirror Based Autostereoscopic Display for Tele-Operation and Telepresence Applications,xe2x80x9d in Stereoscopic Displays and Virtual Reality Systems VII, Proceedings of SPIE, Volume 3957 (McKay, Mair, Mason, Revie) which uses a stretchable membrane mirror. However, the apparatus disclosed in the McKay article has limited field of view, due to the use of conventional projection optics and due to dimensional constraints that limit membrane mirror curvature.
Curved mirrors have also been used to provide real images in stereoscopic systems, where the curved mirrors are not used as collimators. Such systems are disclosed in U.S. Pat. Nos. 4,623,223 (Kempf); and 4,799,763 (Davis et al.) for example. However, systems such as these are generally suitable where only a small field of view is needed.
Notably, existing solutions for stereoscopic projection, such as the system disclosed in U.S. Pat. No. 5,671,992 noted above, project images as real images. However, presentation of a real image has inherent drawbacks for stereoscopic viewing. In a real image display, the image viewed by an observer is formed on or projected onto a display surface. The dimensions of this display surface necessarily limit the field of the projected image. This is true even where the image is then projected on a curved surface. This can result in undesirable distortion and other image aberration, not only constraining field of view, but also limiting image quality overall. Screen artifacts can further degrade image quality in a displayed real image.
As an alternative to real image projection, an optical system can produce a virtual image display. In contrast to methods for forming a real image, a virtual image is not formed on a display surface. That is, if a display surface were positioned at the perceived location of a virtual image, no image would be formed on that surface. Virtual image display has a number of inherent advantages, as is outlined in U.S. Pat. No. 5,625,372 (Hildebrand et al.) As one significant advantage for stereoscopic viewing, the size of a virtual image is not limited by the size or location of a display surface. Additionally, the source object for a virtual image may be small; a magnifying glass, as a simple example, provides a virtual image of its object. Thus, it can be seen that, in comparison with prior art systems that project a real image, a more realistic viewing experience can be provided by forming a virtual image that is disposed to appear some distance away. Providing a virtual image also obviates any need to compensate for screen artifacts, as may be necessary when projecting a real image.
From an optical perspective, it can be seen that there are advantages to autostereoscopic design using pupil imaging. A system designed for pupil imaging must provide separate images, as a stereoscopic pair, to the left and right pupils correspondingly, and must provide natural viewing conditions without goggles or special headgear. In addition, it would be advantageous for such a system to provide the largest possible pupils to the observer, so as to allow some freedom of movement, and to provide an ultra-wide field of view. It is recognized in the optical arts that any single one of these requirements, taken by itself, can be difficult to achieve. An ideal autostereoscopic imaging system must meet the challenge for both requirements in order to provide a more fully satisfactory and realistic viewing experience. In addition, such a system must provide sufficient resolution for realistic imaging, with high brightness and contrast. Moreover, the requirement that equipment have a small footprint imposes physical constraints on system design. There are also dimensional constraints for interocular separation, so that separate images directed to each eye can be precisely spaced and correctly separated for viewing. It is instructive to note that, using conventional lens design techniques, interocular distance constraints limit the ability to achieve larger pupil diameter at a given ultrawide field by simply scaling the projection lens.
Monocentric imaging systems have been shown to provide significant advantages for high-resolution imaging of flat objects, such as is disclosed in U.S. Pat. No. 3,748,015 (Offner), which teaches an arrangement of spherical mirrors arranged with coincident centers of curvature in an imaging system designed for unit magnification. The monocentric arrangement disclosed in the Offner patent minimizes a number of types of image aberration and is conceptually straightforward, allowing a simplified optical design for high-resolution catoptric imaging systems. A monocentric arrangement of mirrors and lenses is also known to provide advantages for telescopic systems having wide field of view, as is disclosed in U.S. Pat. No. 4,331,390 (Shafer). However, while the advantages of monocentric design for overall simplicity and for minimizing distortion and optical aberrations can be appreciated, such a design concept can be difficult to implement in an immersion system requiring wide field of view and large exit pupil with a reasonably small overall footprint. Moreover, a fully monocentric design would not meet the requirement for full stereoscopic imaging, requiring separate images for left and right pupils.
As is disclosed in U.S. Pat. No. 5,908,300, conventional wide-field projection lenses can be employed as projection lenses in a pupil-imaging autostereoscopic display. However, there are a number of disadvantages with conventional approaches for pupil imaging optics. Wide-angle lens systems, capable of angular fields such as would be needed for effective immersion viewing, would be very complex and costly. Typical wide angle lenses for large-format cameras, such as the Biogon(trademark) lens manufactured by Carl-Zeiss-Stiftung in Jena, Germany for example, are capable of 75-degree angular fields. The Biogon lens consists of seven component lenses and is more than 80 mm in diameter, while only providing a pupil size of 10 mm. For larger pupil size, the lens needs to be scaled in size; however, the large diameter of such a lens body presents a significant design difficulty for an autostereoscopic immersion system, relative to the interocular distance at the viewing position. Costly cutting of lenses so that right- and left-eye assemblies could be disposed side-by-side, thereby achieving a pair of lens pupils spaced consistent with human interocular separation, presents difficult manufacturing problems. Interocular distance limitations constrain the spatial positioning of projection apparatus for each eye and preclude scaling of pupil size by simple scaling of the lens. Moreover, an effective immersion system most advantageously allows a very wide field of view, preferably well in excess of 90 degrees, and would provide large exit pupil diameters, preferably larger than 20 mm.
As an alternative for large field of view applications, ball lenses have been employed for specialized optical functions, particularly miniaturized ball lenses for use in fiber optics coupling and transmission applications, such as is disclosed in U.S. Pat. No. 5,940,564 (Jewell) which discloses advantageous use of a miniature ball lens within a coupling device. On a larger scale, ball lenses can be utilized within an astronomical tracking device, as is disclosed in U.S. Pat. No. 5,206,499 (Mantravadi et al.) In the Mantravadi et al. patent, the ball lens is employed because it allows a wide field of view, greater than 60 degrees, with minimal off-axis aberrations or distortions. In particular, the absence of a unique optical axis is used advantageously, so that every principal ray that passes through the ball lens can be considered to define its own optical axis. Because of its low illumination falloff relative to angular changes of incident light, a single ball lens is favorably used to direct light from space to a plurality of sensors in this application. Notably, photosensors at the output of the ball lens are disposed along a curved focal plane.
The benefits of a spherical or ball lens for wide angle imaging are also utilized in an apparatus for determining space-craft attitude, as is disclosed in U.S. Pat. No. 5,319,968 (Billing-Ross et al.) Here, an array of mirrors direct light rays through a ball lens. The shape of this lens is advantageous since beams which pass through the lens are at normal incidence to the image surface. The light rays are thus refracted toward the center of the lens, resulting in an imaging system having a wide field of view. Another specialized use of ball lens characteristics is disclosed in U.S. Pat. No. 4,854,688 (Hayford et al.) in the optical arrangement of the Hayford et al. patent, directed to the transmission of a 2-dimensional image along a non-linear path, such as attached to headgear for a pilot, a ball lens directs a collimated input image, optically at infinity, for a pilot""s view.
Another use for wide-angle viewing capabilities of a ball lens is disclosed in U.S. Pat. No. 4,124,978 (Thompson), which teaches use of a ball lens as part of an objective lens in binocular optics for night viewing.
With each of the patents described above that disclose use of a ball lens, there are suggestions of the overall capability of the ball lens to provide, in conjunction with support optics, wide field of view imaging. However, there are substantial problems that must be overcome in order to make effective use of such devices for immersive imaging applications, particularly where an electronically processed image is projected. Conventional electronic image presentation techniques, using devices such as spatial light modulators, provide an image on a flat surface. Ball lens performance with flat field imaging would be extremely poor.
There are also other basic optical limitations for immersion systems that must be addressed with any type of optical projection that provides a wide field of view. An important limitation is imposed by the Lagrange invariant. Any imaging system conforms to the Lagrange invariant, whereby the product of pupil size and semi-field angle is equal to the product of the image size and the numerical aperture and is an invariant for the optical system. This can be a limitation when using, as an image generator, a relatively small spatial light modulator or similar pixel array which can operate over a relatively small numerical aperture since the Lagrange value associated with the device is small. A monocentric imaging system, however, providing a large field of view with a large pupil size (that is, a large numerical aperture), inherently has a large Lagrange value. Thus, when this monocentric imaging system is used with a spatial light modulator having a small Lagrange value, either the field or the aperture of the imaging system, or both, will be underfilled due to such a mismatch of Lagrange values. For a detailed description of the Lagrange invariant, reference is made to Modern Optical Engineering, The Design of Optical Systems by Warren J. Smith, published by McGraw-Hill, Inc., pages 42-45.
For the purpose of more accurately providing stereoscopic imaging, where the image intended for the left eye differs at least slightly from the image for the right eye, a number of conventional stereoscopic imaging systems utilize head tracking. Head tracking allows a stereoscopic imaging system to adjust display behavior based on sensed data such as the distance of an observer from the display, head orientation, and similar factors.
As is noted in U.S. Pat. No. 6,162,191 (Foxlin), there are four basic types of head tracking technologies, namely optical, mechanical, ultrasonic, and magnetic. With respect to conventional stereoscopic display systems, optical head tracking methods are most widely used. As one example, U.S. Pat. No. 6,011,581 (Swift et al.) discloses use of miniature cameras for eye and head imaging, coupled to an orientation computing subsystem that analyzes images of the observer in order to compute head distance and angle. Other examples of head tracking for stereoscopic display systems include the following:
U.S. Pat. No. 6,163,336 (Richards) discloses ahead tracking system for stereoscopic viewing that further provides eye-tracking in 3 dimensions, using infrared light and reflection techniques.
U.S. Pat. No. 6,075,557 (Holliman et al.) discloses ahead tracking system used to adjust viewing windows for an observer who may be located at one of a number of different positions.
U.S. Pat. No. 6,055,013 (Woodgate et al.) discloses a system that provides discrete viewing zones for proper 3-D display.
European Pat. Application 0 576 106 A1 (Eichenlaub) discloses observer head tracking for a stereoscopic system using a screen display.
U.S. Pat. No. 5,777,720 (Shapiro et al.) discloses observer tracking and a method of calibration for increasing tracking accuracy. Notably, the Shapiro et al. patent also discloses a method of sensing interocular distance, which can vary between one observer and another.
European Pat. Application 0 656 555 A1 (Woodgate et al.) discloses a stereoscopic imaging display with head tracking in three dimensions and with interocular distance sensing. The Woodgate et al. application also discloses methods for tracking and interocular compensation in a stereoscopic display apparatus.
European Pat. Application 0 350 957 (Tomono et al.) discloses an eye tracking method that detects reflected light from a viewer""s face at two different wavelengths to determine the position of pupils and gaze direction.
U.S. Pat. No. 6,069,649 (Hattori) discloses head tracking and compensation in a system using a time-interlaced display screen.
The capability for accurate head tracking, as is disclosed in the above patents, enables a stereoscopic imaging system to suitably adapt image presentation and focus to suit sensed attributes including observer distance, interocular distance, observer gaze point, posture, gesture, and the like. However, although conventional stereoscopic imaging systems can detect head distance, interocular distance, and other features related to stereoscopic imaging, these systems are constrained with respect to their ability to respond to these sensed attributes. For example, even though a system such as that disclosed in U.S. Pat. No. 5,777,720 is able to detect interocular dimension differences between one observer and the next, such a system is limited in its capability to compensate for such variation.
For real images, as provided by the stereoscopic displays in patents cited above, compensation for horizontal distance changes is performed by varying the width and placement of vertical segments of a projected display in some fashion. Compensation for head distance from the display can be compensated by adjusting the tilt angle of a screen or light source, or by moving individual right-eye and left-eye projection apparatus or other optical component. Moreover, the methods described in each of the patents listed above may work acceptably for types of stereoscopic systems that project a real image. However, the conventional methods described for such systems would be awkward and difficult to adapt to a system that provides a virtual image using pupil imaging. Thus it can be seen that there is a need for a suitable head tracking and response mechanism in an autostereoscopic display apparatus that utilizes ball lenses and takes advantage of the benefits of an optically monocentric design.
A related application for head tracking and compensation in an autostereoscopic viewing system is commonly assigned application xe2x80x9cAn Image Display System with Body Position Compensationxe2x80x9d Ser. No. 09/766,899, filed Jan. 22, 2001, in the names of Ronald S. Cok and Mark E. Bridges. This application discloses a chair designed to compensate for viewer head movement in order to maintain proper pupil position. As is noted in this application, compensation by chair movement may be optimal for an initial coarse positioning of an observer. Then, where there are subsequent subtle changes in head movement, it may be more efficient and faster to adjust the position of system optics rather than to reposition a chair. Thus, a combination of controlled movements may be needed to achieve optimal response compensation for positioning of view pupils in an autostereoscopic imaging system.
It is well known to those skilled in the virtual reality art that, while the visual display is the primary component needed for an effective immersion experience, there is substantial added value in complementing visual accuracy with reinforcement using other senses of an observer. While the addition of auditory, tactile, and motion stimuli has been implemented for a more realistic and compelling motion picture experience to an audience, there is a need to provide additional sense stimuli in an autostereoscopic viewing system. Moreover, the use of such additional stimuli may be optimized using sensed feedback information about an observer. Thus it can be seen that, while there are some conventional approaches that meet some of the requirements for stereoscopic imaging, there is a need for an improved autostereoscopic imaging solution for viewing electronically generated images, where the solution provides a structurally simple apparatus, minimizes aberrations and image distortion, and meets demanding requirements for providing wide field of view with large pupil size, for compensating for observer head movement and interocular distance differences, and for providing additional sensory stimulation.
It is an object of the present invention to provide an autostereoscopic image display apparatus capable of modifying the spatial position of a left eye viewing pupil and a right eye viewing pupil in response to feedback data about an observer, said display apparatus comprising:
(a) an adaptive autostereoscopic image delivery system comprising:
(1) an image generator for generating, from an image source, a left eye image and a right eye image;
(2) a control logic processor for of accepting feedback data about said observer and providing a command in response to said feedback data;
(3) a left viewing pupil forming apparatus for providing said left eye image, as a virtual image, to said left viewing pupil for viewing by said observer;
(4) a right viewing pupil forming apparatus for providing said right eye image, as a virtual image, to said right viewing pupil for viewing by said observer,
wherein said left viewing pupil forming apparatus and said right viewing pupil forming apparatus adjust the position of the left viewing pupil and right viewing pupil respectively, based on said command from said control logic processor;
(b) at least one observer feedback sensor for providing said feedback data to said control logic processor.
In an alternate embodiment, the present invention provides left-eye and right-eye images as real images, projected onto a substantially retroreflective screen.
A feature of the present invention is the use of a monocentric arrangement of optical components, thus simplifying design, minimizing aberrations and providing a wide field of view with large exit pupils.
A further feature of the present invention is the use of ball lenses in the left-eye and right-eye projection apparatus, wherein the ball lenses are used to project an image having a wide field of view.
An additional of the present invention is a head-tracking subsystem disposed within a feedback loop in order to respond to sensed position of the observer""s head.
It is an advantage of the present invention that it provides an autostereoscopic display system capable of adapting to variation in observer positioning and movement. In addition, the present invention also adapts to differences in interocular distance, conditioning the display for optimum viewing by each individual observer.
It is a further advantage of the present invention that it allows viewing of a stereoscopic display with pupil imaging, providing a natural, three-dimensional display without requiring wearing of a helmet, glasses, or other object.
It is an advantage of the present invention that it provides a system for wide field stereoscopic projection that is inexpensive when compared with the cost of conventional projection lens systems.
It is yet a further advantage of the present invention that it provides an exit pupil of sufficient size to allow for non-critical alignment of an observer in relation to the display.